In-vehicle camera system and image processing apparatus

ABSTRACT

Automatic exposure control of an in-vehicle camera is performed under dark driving environments such as at night. An in-vehicle camera system includes a vehicle camera mounted in a vehicle configured to capture surroundings of the vehicle, and control circuitry that controls an exposure level of an image captured by the vehicle camera, the control of the exposure level being based on brightness information of a detection area set within the image, the detection area being a portion of the captured image and configured to output the image having exposure control performed thereon to a display.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 15/575,572, filed on Nov. 20, 2017, which isNational Stage Entry of Patent Application No. PCT/JP2016/002203, filedon Apr. 26, 2016, which claims priority from Japanese Patent ApplicationNo. JP 2015-250573, filed Dec. 22, 2015 and Japanese Patent ApplicationNo. JP 2015-114084 filed on Jun. 4, 2015, the entire contents of whichare incorporated by reference.

TECHNICAL FIELD

The present technology relates to an in-vehicle camera system thatperforms exposure control on a camera mounted in a vehicle, and an imageprocessing apparatus that performs exposure control based on images shotby the in-vehicle camera.

BACKGROUND ART

An increasing number of automobiles have recently been equipped withcameras. Images shot by the in-vehicle camera can be used to providedriving support and visual field support. For example, based on theimages shot by the in-vehicle camera, the location and direction of thevehicle can be detected and used for guidance information, and oncomingvehicles, preceding vehicles, pedestrians, obstacles, and others can bedetected for accident prevention. In addition, during night-timedriving, the images shot by the in-vehicle camera can be processed tosense the headlights of the oncoming vehicle (or following vehicle) andthe taillights of the preceding vehicle and detect information on theother surrounding vehicles (for example, refer to PTLs 1 and 2).

CITATION LIST Patent Literatures

[PTL 1]

JP 2005-92857 A

[PTL 2]

JP 2010-272067 A

[PTL 3]

JP 2004-23226 A

[PTL 4]

JP 2013-205867 A

SUMMARY Technical Problem

It is desirable to provide an in-vehicle camera system that performsexposure control on a camera mounted in a vehicle, and an imageprocessing apparatus that performs an exposure control based on imagesshot by the in-vehicle camera.

Solution to Problem

The technique disclosed in the present specification was made in view ofthe above problem, and a first aspect of the present technology providesan in-vehicle camera system including: a vehicle camera mounted in avehicle configured to capture surroundings of the vehicle, and controlcircuitry configured to control an exposure level of an image capturedby the vehicle camera, the control of the exposure level being based onbrightness information of a detection area set within the image, thedetection area being a portion of the captured image and configured tooutput the image having exposure control performed thereon to a display.

The term “system” here refers to a logical assembly of a plurality ofdevices (or functional modules implementing specific functions),regardless of whether the devices or functional modules are included ina single housing.

According to a second aspect of the technology there is provided animage processing apparatus, including: control circuitry configured tocontrol an exposure level of an image captured by a vehicle camera, thecontrol of the exposure level being based on brightness information of adetection area set within the image, the detection area being a portionof the captured image and configured to output the image having exposurecontrol performed thereon to a display.

According to a third aspect of the technology there is provided an imageprocessing method, including: controlling, using control circuitry, anexposure level of an image captured by a vehicle camera, the control ofthe exposure level being based on brightness information of a detectionarea set within the image, the detection area being a portion of thecaptured image, and outputting the image having exposure controlperformed thereon to a display.

Advantageous Effects of Invention

According to an embodiment of the technology disclosed herein, it ispossible to provide an in-vehicle camera system that performs exposurecontrol on a camera mounted in a vehicle, and an image processingapparatus that performs exposure control based on images shot by thein-vehicle camera.

The advantageous effects described herein are mere examples, and theadvantageous effects of the present technology are not limited to them.In addition, the present technology may have additional advantageouseffects other than the ones described above.

Other objectives, features, and advantages of the technology disclosedherein will be clarified by more detailed descriptions of the followingembodiments and the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example in which cameras are mountedinside the interior of an automobile 100.

FIG. 2 is a diagram illustrating an example in which cameras are mountedoutside the interior of an automobile 200.

FIG. 3 is a diagram illustrating a scene ahead of or behind the vehicle(at night or in a tunnel).

FIG. 4 is a diagram illustrating an image 401 of a scene under darkdriving environments shot by the in-vehicle camera at a (long) exposuretime for low illumination as illustrated in FIG. 3.

FIG. 5 is a diagram illustrating an image 501 of a scene under darkdriving environments shot by the in-vehicle camera at a (short) exposuretime for high illumination as illustrated in FIG. 3.

FIG. 6 is a diagram illustrating a configuration example of anin-vehicle camera system 600.

FIG. 7 is a flowchart of a process for automatic exposure control of thein-vehicle camera.

FIG. 8 is a diagram for describing a positional relationship between adistant road vanishing point and a reference point in the drivingdirection of the vehicle.

FIG. 9 is a diagram for describing a positional relationship between adistant road vanishing point and a reference point in the drivingdirection of the vehicle.

FIG. 10 is a diagram for describing a positional relationship between adistant road vanishing point and a reference point in the drivingdirection of the vehicle.

FIG. 11 is a diagram for describing a positional relationship between adistant road vanishing point and a reference point in the drivingdirection of the vehicle.

FIG. 12 is a diagram for describing a positional relationship between adistant road vanishing point and a reference point in the drivingdirection of the vehicle.

FIG. 13 is a diagram for describing a positional relationship between adistant road vanishing point and a reference point in the drivingdirection of the vehicle.

FIG. 14 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

FIG. 15 is a diagram illustrating a scan line 1501 at one vertical pixelposition v within a distant small frame 1500.

FIG. 16 is a diagram illustrating a brightness distribution on one scanline (horizontal direction) within a distant small frame.

FIG. 17 is a diagram illustrating weights in individual scan lines(vertical direction).

FIG. 18 is a diagram illustrating a brightness horizontal profile withina distant small frame.

FIG. 19 is a flowchart of a process for determining whether to shift toa vehicle light source detection mode.

FIG. 20 is a diagram for describing a method for approximating a distantroad vanishing point on a curve road.

FIG. 21 is a diagram for describing a method for approximating a distantroad vanishing point on a road with a change in inclination.

FIG. 22 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of the subject vehicle.

FIG. 23 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of the subject vehicle.

FIG. 24 is a flowchart of a process for defining a distant small framefor prioritizing detection of the lamps of an automobile.

FIG. 25A is a flowchart of a process for creating a brightnesshorizontal profile within a distant small frame.

FIG. 25B is a flowchart of the process for creating a brightnesshorizontal profile within a distant small frame.

FIG. 26A is a flowchart of another process for creating a brightnesshorizontal profile within a distant small frame.

FIG. 26B is a flowchart of the other process for creating a brightnesshorizontal profile within a distant small frame.

FIG. 27 is a diagram illustrating adjacent maximum values and a minimumvalue between the maximum values in a brightness horizontal profilewithin a distant small frame.

FIG. 28A is a flowchart of a process for determining whether the lampsof an automobile are located within a distant small frame based on abrightness horizontal profile.

FIG. 28B is a flowchart of the process for determining whether the lampsof an automobile are located within a distant small frame based on abrightness horizontal profile.

FIG. 29 is a flowchart of a process for performing automatic exposurecontrol of the in-vehicle camera based on the maximum values and theminimum value in the brightness horizontal profile.

FIG. 30A is a diagram illustrating a brightness horizontal profilewithin a distant small frame when an evaluation value F for automaticexposure control is large.

FIG. 30B is a diagram illustrating a brightness horizontal profilewithin a distant small frame when the evaluation value F for automaticexposure control is small.

FIG. 31 is a diagram illustrating a brightness horizontal profile withina distant small frame (in the case of shooting in an exposure conditionoptimized for luminous point separation).

FIG. 32 is a diagram illustrating a brightness horizontal profile withina distant small frame (in the case of shooting in an exposure conditioninsufficient for luminous point separation).

FIG. 33 is a flowchart of a process for performing automatic exposurecontrol of the in-vehicle camera optimized for luminous point separationbased on the shape of the brightness horizontal profile within a distantsmall frame.

FIG. 34 is a diagram illustrating a headlight shape 3401 when afollowing vehicle or an oncoming vehicle is shot by the in-vehiclecamera in the scene under dark driving environments.

FIG. 35 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

FIG. 36 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

FIG. 37 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

FIG. 38 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

FIG. 39 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

FIG. 40 is a diagram illustrating an example in which a distant smallframe is defined based on a distant road vanishing point in the drivingdirection of a subject vehicle.

DESCRIPTION OF EMBODIMENTS

Embodiments of the technology disclosed herein will be described belowin detail with reference to the drawings.

FIG. 1 illustrates an example in which cameras are mounted in anautomobile (standard-sized passenger car) 100. Referring to FIG. 1, twocameras 101 and 102 are mounted inside the interior of the automobile100. One camera 101 has an imaging direction oriented to the area aheadof the automobile 100 (driving direction), and is placed near the frontwindow (room mirror), for example. The other camera 102 has a shootingdirection oriented toward the area behind the automobile 100, and isplaced near the rear window at the back of the interior, for example. Inthe example illustrated in FIG. 1, the other camera 102 is arranged inthe vehicle interior to shoot the area behind the automobile 100 throughthe rear window (alternatively, the area behind the automobile 100 maybe shot by a camera 202 arranged outside the vehicle, as illustrated inFIG. 2 (described later)). The viewing angles of the cameras 101 and 102are indicated with their respective reference numbers 111 and 112. Thecamera 101 can shoot a preceding vehicle or an oncoming vehicle in thearea ahead of the subject vehicle in the driving direction. The camera102 can also shoot a following vehicle in the area behind the subjectvehicle in the driving direction. The cameras 101 and 102 can be fixedby any method to their positions in the vehicle.

FIG. 2 illustrates another example in which cameras are mounted in anautomobile (standard-sized passenger car) 200. Referring to FIG. 2, fourcameras 201 to 204 are mounted outside the interior of the automobile200. The camera 201 is placed at the front end of the automobile 200(for example, near the front grill or the front bumper cover) to shootthe area ahead of the subject vehicle (in the driving direction). Thecamera 202 is placed at the rear end of the automobile 200 (for example,near the rear bumper cover) to shoot the area behind the subjectvehicle. The cameras 203 and 204 are placed near the leading ends ofright and left door mirrors with shooting directions oriented to thearea behind the automobile 200. The cameras 203 and 204 may be used incombination with the right and left door mirrors. Alternatively, thecameras 203 and 204 may replace the right and left door mirrors. Theviewing angles of the cameras 201 to 204 are indicated with theirrespective reference numbers 211 to 214. The camera 201 can shoot apreceding vehicle and an oncoming vehicle in the area ahead of thesubject vehicle in the driving direction. The cameras 202 to 204 canalso shoot a following vehicle in the area behind the subject vehicle inthe driving direction. The cameras 201 to 204 can be fixed by any methodto their positions in the vehicle.

The technology disclosed herein is based on the assumption that one ormore cameras (in-vehicle cameras) are mounted in an automobile. Theplacement positions of the in-vehicle cameras illustrated in FIGS. 1 and2 are mere examples. The in-vehicle cameras may be placed at positionsother than the foregoing ones in the vehicle. Two or more in-vehiclecameras may be combined. The vehicles in which the in-vehicle camerasare mounted are not limited to standard-sized passenger vehicles but maybe medium-sized vehicles, large-sized vehicles, or two-wheel vehiclessuch as motorcycles.

Under dark driving environments such as at night or in a tunnel, imagesshot by the in-vehicle camera can be processed to sense the headlightsof a following vehicle (or an oncoming vehicle) and the taillights of apreceding vehicle and detect information on the other surroundingvehicles (as described above). For example, it is necessary to check thedistant following vehicle to change lanes on an expressway. However, itis difficult to recognize clearly the shape of the surrounding vehiclein dark conditions such as at night. In general, a driver perceivesvisually the status of approaching of the surrounding vehicle throughits right and left lights or the like. Accordingly, it is necessary tocontrol the camera to check visually the separation of the right andleft light sources.

The exposure control technology applied to normal (or common) photoshooting or in-room video shooting is basically intended to subject theentire screen to average exposure control. For example, there isexposure control method by which the screen is simply divided into aplurality of sections as brightness detection frames, and the sums ofincident light in the individual detection frames are integrated toobtain the amount of light, and the exposure of the camera isautomatically controlled based on the amount of light (for example,refer to PTL 3). In addition, there is exposure control method by whichthe amounts of light are weighted and integrated in the individualdetection frames for the optimum light exposures for individual scene,and automatic exposure and gain adjustment of the camera are performedsuited to the actual scenes.

However, when the in-vehicle camera shots a following or precedingvehicle (hereinafter, referred to as “distant vehicle”) at apredetermined distance L (for example, L=about 150 m) from the subjectvehicle, the proportion of the headlights (or taillights) as lightsources of the distant vehicle in the viewing field of the in-vehiclecamera is small. Accordingly, the application of the method by whichexposure control is averagely performed on the entire screen would leadto a (long) light exposure time for low illumination that is likely tocause overexposure. For example, when the screen is simply divided intoa plurality of regions as detection frames and the brightness values inthe detection frames are weighted, the proportion of the noticed vehiclein the detection frame is too small. Therefore, even though theapplicable detection frame is weighted increasingly, the expression ofaverage brightness of the screen is prioritized to make it difficult toperform a stable automatic exposure control.

The case in which the in-vehicle camera shots a scene ahead of or behindthe vehicle (viewable on a door mirror or a room mirror) under darkdriving environments such as at night or in a tunnel as illustrated inFIG. 3 will be discussed. It is assumed that, in the scene illustratedin FIG. 3, a following vehicle 301 is driving on a road (left lane) 302behind the subject vehicle. Under dark driving environments, thefollowing vehicle 301 has the two right and left headlights turned on.

FIG. 4 illustrates an image of the scene under dark driving environmentsas illustrated in FIG. 3 shot by the in-vehicle camera at a (long) lightexposure time for low illumination. Since the headlights of thefollowing vehicle 301 are of high illumination, overexposure is causedto saturate even peripheral pixels in white as indicated with referencenumber 401. In this case, it is difficult to catch the headlights as twoluminous points. When the normal automatic light exposure control isapplied to the in-vehicle camera, the in-vehicle camera tends to shootat a (long) light exposure time for low illumination, and as a result,overexposure as illustrated in FIG. 4 is likely to occur.

FIG. 5 illustrates an image of a scene under dark driving environmentsas illustrated in FIG. 3 shot by the in-vehicle camera at a (short)exposure time for high illumination. By suppressing the light exposure,it is possible to catch the right and left high-illumination headlightsof the following vehicle 301 as two luminous points as indicated withreference sign 501.

As illustrated in FIG. 34, when a following vehicle or an oncomingvehicle is shot by the in-vehicle camera under dark drivingenvironments, the lateral width of a headlight shape 3401 is designatedas Wh, the average of lateral brightness values of the headlights as Yh,the narrowest vertical width of the headlight shape 3401 as Wv, and theaverage of brightness values at the narrowest portion as Yv. Theapplicant of the subject application supposes that Wh·Yh/Wv·Yv becomesabout 2 or 3 as expressed by the following mathematical expression (1)(A denotes a constant of about 2 or 3).

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 1} \right\rbrack & \; \\{\frac{W\;{h \cdot {Yh}}}{{Wv} \cdot {Yv}} > A} & (1)\end{matrix}$

By controlling the exposure time as described above, the user (driver)can estimate the width of the following vehicle 301 and the distancefrom the following vehicle 301 from the two luminous points and easilydetermine whether to change lanes even when the vehicle is equipped withan electronic mirror in which a room mirror or side mirrors are replacedwith a display. In addition, by applying the foregoing exposure controlto the front camera 101, it is possible to determine the width of thepreceding vehicle 302 or the oncoming vehicle from the two luminouspoints. With information on the angle of view of the headlights of thefollowing vehicle 301, it is possible to calculate the distance betweenthe subject vehicle and the following vehicle 301 based on theprinciples of triangulation. It is further possible to calculate thedistance from an oncoming vehicle from the two luminous points of theheadlights of the oncoming vehicle. The foregoing information can beapplied to a headlight control system in which the headlights of thesubject vehicle are automatically switched between high beam and lowbeam according to the distance from the oncoming vehicle, for example.

For example, when being driving on an expressway, the subject vehiclemay speedily catch up with a preceding vehicle and be caught by afollowing vehicle. Accordingly, distant visual field support isimportant for safe lane change. The in-vehicle camera related to drivingsupport and distant visual field support needs to catch more correctlydistant vehicles possibly approaching the subject vehicle. However,under dark driving environments such as at night or in a tunnel, thereis a problem that overexposure is likely to occur when the lamps ofhigh-illumination (backlight) surrounding vehicles are shot by thein-vehicle camera as described above. In particular, the proportion ofthe headlights of the distant vehicle as light sources in the viewingfield is small. Accordingly, when the normal automatic light exposurecontrol is applied to perform an average exposure control on the entirescreen, overexposure is likely to occur to saturate even peripheralpixels in white, and it is difficult to catch the headlights as twoluminous points.

On a general road in town, other vehicles hardly approach the subjectvehicle from a long distance. It is thus not necessarily desired toprioritize uniformly visual check of light sources of the distantvehicles. Therefore, switching between exposure control modes isnecessary depending on purposes.

The exposure control mode may be determined depending on changes inenvironmental light sources under driving environments such as urbanareas, tunnels, nighttime, roads without street lights, expressways, anddusk, and average vehicle velocity on roads.

Accordingly, exposure control technology for preferably shooting adistant vehicle with high-illumination headlights illuminated by thein-vehicle camera without occurrence of overexposure under dark drivingenvironments such as at night or in a tunnel will be proposed herein.Briefly, the technology disclosed herein enables optimum exposurecontrol by performing a waveform profile analysis of the distant vehiclewith headlights illuminated at night within a small frame set in theimage shot by the camera to make the headlights of the distant vehicleeasy to view.

FIG. 6 illustrates schematically a configuration example of anin-vehicle camera system 600 that enables automatic exposure control ofthe in-vehicle camera.

The target vehicle is equipped with a plurality of sensors such as animage sensor 601, a yaw angle detection unit 602, a pitch angledetection unit 603, and a GPS (global positioning system) sensor 604.

The image sensor 601 is equivalent to an in-vehicle camera mounted inthe subject vehicle for the purpose of shooting the surrounding vehiclesand others, and is placed at any of the positions illustrated in FIG. 1or 2, for example. The image sensor 601 may include two or morein-vehicle cameras. Alternatively, the image sensor 601 may be a sensorother than an in-vehicle camera.

The yaw angle detection unit 602 detects the rotational angle θ of thesubject vehicle around the yaw. For example, when the subject vehicle isdriving around a curve, the yaw angle detection unit 602 detectsmomentary changes δθ in the rotational angle of the subject vehiclearound the yaw equivalent to the turning angle of the road. The yawangle detection unit 602 is composed of a yaw rate sensor and a rudderangle sensor, for example. The yaw angle θ detected by the yaw angledetection unit 602 is equivalent to the turning angle of the subjectvehicle relative to the straight-running direction of the subjectvehicle, for example.

The pitch angle detection unit 603 detects the rotational angle ϕ andthe angular velocity of the subject vehicle related to the pitch. Forexample, when the subject vehicle is driving on an uphill or downhillroad, the pitch angle detection unit 603 detects momentary changes δϕ inthe inclination of the slope. The pitch angle detection unit 603 iscomposed of a pitch rate sensor and a gyro sensor, for example. Thedetected pitch angle ϕ is equivalent to the inclination of the subjectvehicle driving on an uphill or downhill road, the inclination of thevehicle body resulting from a freight or the like in the vehicleinterior, or the inclination of the vehicle body resulting from aninertial force due to the acceleration or deceleration of the subjectvehicle, for example.

The GPS sensor 604 receives radio waves (GPS signals) from overhead GPSsatellites. Position determination is performed based on GPS signalsfrom four GPS satellites to determine the receipt time and the receivercoordinates (a point in a three-dimensional space) at the same time (asis well known).

A processing unit 605 includes a memory 606 used as a working area orfor storing data and programs in a non-volatile manner to executevarious processes. For example, the processing unit 605 executes aprocess for controlling an automatic exposure in the in-vehicle camera.The processing unit 605 also performs processes for recording a drivinghistory of the subject vehicle, determining the driving direction, andcontrolling the mode transition in the automatic exposure control, basedon detection signals from the image sensor 601, the yaw angle detectionunit 602, the pitch angle detection unit 603, and the GPS sensor 604.The memory 606 includes ROM (read only memory) storing in advanceinformation for use in the automatic exposure control, such as weightinginformation (described later) and various setting values.

A setting input unit 607 is composed of a touch panel or operators suchas buttons or switches, for example, to accept input operations from theuser (for example, the driver of the subject vehicle) for executing theprocesses with the processing unit 605. The user can make settings formode transition in the automatic exposure control of the in-vehiclecamera via the setting input unit 607.

An image output unit 608 and a sound output unit 609 output the resultsof execution of the processes by the processing unit 605 by an image ora sound, respectively. The image output unit 608 has a monitor screenobservable from the driver's seat of the subject vehicle, for example,to notify visually a mode shift in the automatic exposure control of thein-vehicle camera and display the results of detection of a distantvehicle. The sound output unit 609 is composed of a speaker, forexample, to notify by a sound a mode shift in the automatic exposurecontrol of the in-vehicle camera and the results of detection of adistant vehicle.

FIG. 7 represents a flowchart of a process for the automatic exposurecontrol of the in-vehicle camera executed by the in-vehicle camerasystem 600 illustrated in FIG. 6. The automatic exposure controlillustrated in the drawing is started by the processing unit 605 whenthe in-vehicle camera (image sensor 601) is powered on.

First, the processing unit 605 captures an image of the entire screenshot by the in-vehicle camera and subjects the image to raster scanning(step S701).

Next, the processing unit 605 performs a normal automatic exposurecontrol on the in-vehicle camera (step S702). The exposure controltechnology applied to the normal (or common) photo shooting or in-roomvideo shooting is basically intended to subject the entire screen toaverage exposure control. For example, there is exposure control methodby which the screen is simply divided into a plurality of sections asbrightness detection frames, and the sums of incident light in theindividual detection frames are integrated to obtain the amount oflight, and the exposure of the camera is automatically controlled basedon the amount of light (for example, refer to PTL 3).

Next, the processing unit 605 determines whether to shift to a vehiclelight source detection mode (step S703). The vehicle light sourcedetection mode here refers to an operation mode in which the presence orabsence of the lamps of an automobile (the headlights or taillights of adistant vehicle) is detected under dark driving environments such as atnight or in a tunnel. The detection of the vehicle light sources isperformed only within a distant small frame described later. To shift toa luminous point separation optimized automatic exposure control mode(described later) that is optimized such that two luminous points of adistant vehicle can be visually separated (or exposure control isperformed with priority given to the separation of two luminous pointsof a distant vehicle), the vehicle light source detection is performedin advance within the distant small frame. The details of a process fordetermining whether to shift to the vehicle light source detection modewill be described later.

When determining not to shift to the vehicle light source detection mode(step S703: No), the processing unit 605 returns to step S701 to performcontinuously the normal automatic exposure control on the in-vehiclecamera.

Meanwhile, when determining to shift to the vehicle light sourcedetection mode (step S703: Yes), the processing unit 605 then performs aprocess for correcting a distant road vanishing point in the drivingdirection of the subject vehicle within the image shot by the in-vehiclecamera (step S704).

The vanishing point refers to a point on which straight lines convergein an image corresponding to parallel lines in a three-dimensional spacethat are projected onto the image by perspective transformation (forexample, refer to PTL 4). The vanishing point is a theoretical distantpoint at infinity. The term “vanishing point” may recall two kinds ofpoints: a vanishing point in an image shot by the in-vehicle camera(distant point at infinity) and a vanishing point at which ridge linesof the road cross. For the sake of convenience, the vanishing point(distant point at infinity) in the image shot by the in-vehicle camerawill be here called “reference point,” and the vanishing point at whichthe ridge lines of the road cross will be here called “distant roadvanishing point in the driving direction” (or simply “vanishing point”).The reference point in the image shot by the in-vehicle camera is afixed value determined by camera parameters or the like, and is storedin advance in the memory (ROM) 606. FIG. 8 illustrates the relationshipbetween the distant road vanishing point 802 in the driving direction ofthe subject vehicle and the reference point 801, when shooting the areabehind the subject vehicle driving on a straight road by the in-vehiclecamera mounted in the subject vehicle. For example, to shoot the areabehind the subject vehicle driving on a straight road by the in-vehiclecamera mounted in the subject vehicle, in the image captured by thecamera with an optical axis parallel to the road as illustrated in FIG.8 in the driving direction of the vehicle, there occurs an overlapbetween the vanishing point (distant point at infinity) in the imageshot by the in-vehicle camera and a road parallel infinite distance atwhich the ridge lines of a driving road 803 cross at a distance, and thevanishing points almost coincide with each other. However, depending onthe position and angle at which the in-vehicle camera is attached to thesubject vehicle, the reference point 801 and the distant road vanishingpoint 802 in the driving direction are shifted from each other in somedegree.

As illustrated in FIG. 9, when the area behind the subject vehicle isshot by the in-vehicle camera mounted in the subject vehicle at the timeof passage through a right-hand curve road 903, a distant road vanishingpoint 902 in the driving direction is shifted leftward with respect to areference point 901 (or when the area ahead of the subject vehicle isshot by the in-vehicle camera at the time of passage through a left-handcurve, a distant road vanishing point in the driving direction is alsoshifted leftward with respect to a reference point).

In contrast, when the area behind the subject vehicle is shot by thein-vehicle camera mounted in the subject vehicle at the time of passagethrough a left-hand curve road 1003, a distant road vanishing point 1002in the driving direction is shifted rightward with reference to thereference point 1001 (or when the area ahead of the subject vehicle isshot by the in-vehicle camera mounted in the subject vehicle at the timeof passage through a left-hand curve road 1003, a distant road vanishingpoint in the driving direction is also shifted rightward with respect toa reference point) as illustrated in FIG. 10.

The processing unit 605 can detect to what degree the road on which thesubject vehicle is driving have curved leftward or rightward, based on adriving history including the yaw angle information detected by the yawangle detection unit 602. Otherwise, the processing unit 605 can predictto what degree the road on which the subject vehicle is driving willcurve leftward or rightward from now on, based on current positioninformation and map information obtained by the GPS sensor 604.

When there is a change in the inclination of the road on which thesubject driving is driving, the distant road vanishing point in thedriving direction may be shifted vertically with respect to thereference point. FIG. 11 illustrates a profile of the road on which thesubject vehicle is driving (the left-to-right direction in the drawingis the driving direction of the subject vehicle). Reference number 1101indicates a profile of a flat (or uniformly inclined) road, referencenumber 1102 indicates a profile of a road with a downhill ended at aspot 1110, and a reference number 1103 denotes a profile of a road withan uphill ended at the spot 1110.

During driving on a flat (or uniformly inclined) road as indicated withreference number 1101, when the area behind the subject vehicle is shotby the in-vehicle camera mounted in the subject vehicle, the distantroad vanishing point in the driving direction almost coincides with thevanishing point (distant point at infinity) in the image shot by thein-vehicle camera as illustrated in FIG. 8.

Meanwhile, when the area at the back side of a road 1203 is shot by thein-vehicle camera mounted in the subject vehicle at the spot 1110 at theend of the downhill as indicated with reference number 1102, the distantroad vanishing point 1202 in the driving direction is shifted upwardwith reference to a reference point 1201 as illustrated in FIG. 12 (orwhen the area at the front side of the road 1203 is shot by thein-vehicle camera at the beginning of an uphill, a distant roadvanishing point in the driving direction is also shifted upward withreference to a reference point).

When the area at the back side of a road 1303 is shot by the in-vehiclecamera mounted in the subject vehicle at the spot 1110 at the end of theuphill as indicated with reference number 1103, as illustrated in FIG.13, a distant road vanishing point 1302 in the driving direction isshifted downward with respect to a reference point 1301 (or when thearea at the front side of a road is shot by the in-vehicle camera at thebeginning of a downhill, a distant road vanishing point in the drivingdirection is also shifted downward with reference to a reference point).

As a summary of the descriptions in FIGS. 8 to 13, in the area behindthe subject vehicle, the distant road vanishing point is corrected basedon the direction of the driving path of the subject vehicle, and in thearea ahead of the subject vehicle, the distant road vanishing point iscorrected based on the predicted driving direction of the subjectvehicle (the road surface direction along the road on which the subjectvehicle is driving).

The processing unit 605 can detect changes in the inclination of theroad on which the subject vehicle has driven based on a driving historyincluding the pitch angle information detected by the pitch angledetection unit 603. Otherwise, the processing unit 605 can predict thatan uphill or a downhill starts on the road on which the subject vehicleis driving, based on the current position information and the mapinformation obtained by the GPS sensor 604.

Therefore, at step S704, the processing unit 605 decides the distantroad vanishing point in the driving direction of the subject vehicle,based on the attachment position and attachment angle of the in-vehiclecamera and the driving history (changes in the driving direction of thesubject vehicle obtained by the yaw angle detection unit 602, theinclination of the subject vehicle obtained by the pitch angle detectionunit 603, and the current position information obtained by the GPSsensor 604) (that is, the processing unit 605 corrects the position ofthe distant road vanishing point based on the reference point).

Next, the processing unit 605 defines a distant small frame toprioritize the detection of the lamps of an automobile (the illuminatedheadlights or taillights of a distant vehicle) on the screen after thecorrection of the distant road vanishing point (step S705).

For example, the in-vehicle camera related to a distant viewing fieldsupport on an expressway needs to catch a distant vehicle possiblyapproaching the subject vehicle in a more correct manner. However, theproportion of the headlights of the distant vehicle as light sources inthe field of view is small, and the application of the normal automaticexposure control would easily cause overexposure. This is because thedistant small frame is defined to prioritize the detection of lamps.

It is presumed that the distant vehicle possibly approaching the subjectvehicle is driving in the direction of the driving path of the subjectvehicle in the area behind the subject vehicle, or is driving in thepredicted driving direction of the subject vehicle (the road surfacedirection along the road on which the subject vehicle is driving) in thearea ahead of the subject vehicle. Therefore, it is expected that, bydefining the distant small frame with respect to the distant roadvanishing point in the driving direction of the subject vehicle, thedesired distant vehicle can be caught within the distant small frame.

At step S705, the distant small frame is defined based on the distantroad vanishing point corrected at previous step S704. FIG. 14illustrates an example in which a distant small frame 1401 is definedwith respect to a distant road vanishing point 1402 in the image shot bythe in-vehicle camera. The lamps (the illuminated headlights ortaillights of a distant vehicle) are detected only within the distantsmall frame because the proportion of the headlights or taillights ofthe distant vehicles as light sources in the field of view of thein-vehicle camera is small. The distant small frame 1401 is basicallyset at or near a position under the distant road vanishing point 1402,for the obvious reason that a distant vehicle 1404 is driving on theroad surface under the distant road vanishing point in the shot image.Further, this is also because the streetlights are generally supposed tobe located higher than the camera in the subject vehicle and positionedabove the distant vanishing point in the shot image, and thus thestreetlights are to be excluded from the frame. In addition, bynarrowing down the position and size of the distant small frame 1401 tothe neighborhood of the distant road vanishing point 1402 as describedabove, it is possible to exclude light sources other than the vehicles,such that streetlights 1405 at both sides of a road 1403, from thedetection target.

The size and shape (rectangular, trapezoid, or the like) of the distantsmall frame may be pre-defined. In the following description, the sizeand shape of the pre-defined distant small frame are recorded in advancein the memory (ROM) 606. Alternatively, the position, size, and shape ofthe distant small frame may be changed depending on the position of thedistant road vanishing point because the distant road vanishing pointmoves from the reference point due to curves or undulations in the roadon which the subject vehicle is driving (refer to the foregoingdescriptions and see FIGS. 8 to 13). The details of a process fordefining the distant small frame performed in step S705 will bedescribed later.

Next, the processing unit 605 produces a brightness horizontal profilewithin the distant small frame defined at previous step S705 (stepS706). Specifically, based on the sum (or average) of brightness valuesof the pixels in the scan line direction in the distant small frame,weights in individual scan lines are decided, and the brightness valuesof the pixels in the scan direction are smoothed out by adding theweights in the individual scan lines at individual horizontal pixelpositions within the distant small frame, thereby producing a brightnesshorizontal profile within the distant small frame.

FIG. 15 illustrates a scan line 1501 at one vertical pixel position vwithin a distant small frame 1500. FIG. 16 illustrates a horizontalbrightness distribution within the scan line 1501 at the vertical pixelposition v. In FIG. 16, the transverse axis indicates horizontal pixelposition u within the distant small frame, and the longitudinal axisindicates brightness value Y(u, v) at each horizontal pixel position u.FIG. 17 illustrates weights in individual scan lines (v) (verticaldirection) obtained by summing up the brightness values of the pixels atthe same horizontal position within the distant small frame. In FIG. 17,the longitudinal axis indicates the scan position (vertical pixelposition v) within the distant small frame, and the transverse axisindicates weight W(v) (resulting from the sum of the brightness valuesof the pixels) at each scan position (=ΣY(ui, v)). As illustrated inFIG. 15, when a distant vehicle with illuminated headlights is includedin the distant small frame, it is predicted that the sum of thebrightness values, that is, the weight becomes larger near the scanposition of the headlights. FIG. 18 illustrates a brightness horizontalprofile within the distant small frame obtained by adding the weights inthe individual scan lines to the brightness values of the pixels in thescan direction at the individual horizontal pixel positions (u) withinthe distant small frame. In FIG. 18, the transverse axis indicates thehorizontal pixel position u within the distant small frame, and thelongitudinal axis indicates weighted brightness sum value Ysum(u)obtained by weighting the brightness values of the pixels at thehorizontal pixel position u. The detailed process for producing thebrightness horizontal profile within the distant small frame performedat step S706 will be described later.

Then, the processing unit 605 checks whether there are the lamps of anautomobile (that is, the headlights of a distant oncoming vehicle (orfollowing vehicle) or the tail lights of a preceding vehicle) within thedistant small frame defined at previous step S705 (step S707) based onthe brightness horizontal profile produced at the previous step S706. Asillustrated in FIG. 15, in the distant small frame 1500 including twoluminous points (the headlights of a distant vehicle), it is presumedthat a brightness horizontal profile having two peaks (maximum points)is produced as illustrated in FIG. 18. The detailed procedure of aprocess for checking the presence or absence of the lamps of anautomobile within the distant small frame performed at step S707 will bedescribed later.

When there are no lamps of an automobile within the distant small frame(step S707: No), the processing unit 605 returns to step S701 to performa normal automatic exposure control on the in-vehicle camera. However,the in-vehicle camera has already shifted to the luminous pointseparation optimized automatic exposure control mode (described later).To change to the normal automatic exposure control mode (step S711:Yes), the driver of the subject vehicle is notified in advance of themode change through the image output unit 608 or the sound output unit609 (step S712).

When there are the lamps of an automobile within the distant small frame(step S707: Yes), the processing unit 605 optimizes the automaticexposure control on the in-vehicle camera such that the two luminouspoints (the right and left headlights of an oncoming vehicle (orfollowing vehicle), or the right and left taillights of a precedingvehicle) can be separated in the luminous point separation optimizedautomatic exposure control mode (step S710). However, to change from thenormal automatic exposure control mode to the luminous point separationoptimized automatic exposure control mode (step S708: Yes), the driverof the subject vehicle is notified in advance of the mode change throughthe image output unit 608 or the sound output unit 609 (step S709).

The exposure control modes are not necessarily limited to the normalautomatic exposure control mode and the luminous point separationoptimized automatic exposure control mode, but may include anintermediate mode. For example, exposure control×α in the normalautomatic exposure control mode and exposure control×β in the luminouspoint separation optimized automatic exposure control mode may beperformed (α+β=1) such that α and β are controlled depending on roadenvironments and outside light source environments.

In this embodiment, it is determined at step S707 whether there are thelamps of an automobile within the distant small frame, and then theexposure control is performed at step S710. Alternatively, in reverse,the exposure control may be performed, and then the presence or absenceof the lamps of an automobile within the distant small frame determined.

FIG. 19 indicates a flowchart of a process for determining whether toshift to the vehicle light source detection mode performed at step S703of the flowchart illustrated in FIG. 7.

First, the processing unit 605 checks whether the shift to the vehiclelight source detection mode is enabled (step S1901). The user (forexample, the driver of the subject vehicle) is allowed to make a modesetting through the setting input unit 607, and perform thedetermination process depending on the setting conditions and others.When the shift to the vehicle light source detection mode is not enabled(step S1901: No), the processing unit 605 returns the negative result ofdetermination on the shift to the vehicle light source detection mode(step S1905).

When it is not necessary to detect the lamps of a distant vehicle fromthe image shot by the in-vehicle camera, the driver may disable theshift to the vehicle light source detection through the image sensor 601such that the normal automatic exposure control is continuously appliedto the in-vehicle camera. For example, to shoot not only the distantvehicle but also the entire scenery by the in-vehicle camera, the drivermay disable the shift to the vehicle light source detection.

When the shift to the vehicle light source detection mode is enabled(step S1901: Yes), the processing unit 605 further checks whether thesubject vehicle is currently driving on an expressway (step S1902).

The processing unit 605 can check whether the subject vehicle iscurrently driving on an expressway based on the position informationobtained by the GPS sensor 604. Alternatively, the processing unit 605may determine whether the road on which the subject vehicle is drivingis an expressway through analysis of the image shot by the image sensor601. Alternatively, the processing unit 605 may determine whether thesubject vehicle is currently driving on an expressway based on whetherthe subject vehicle is charged for an expressway toll by an ETC(electronic toll collection system) (for example, the subject vehicleholds entry information received by an in-vehicle ETC machine at anexpressway toll gate antenna).

On an expressway, the subject vehicle and the surrounding vehicles aresupposed to drive at high speeds. The expressway is linear or gentlycurved and can be overlooked. For example, when the subject vehicle isabout to change lanes on an expressway under dark driving environmentssuch as at night or in a tunnel, a distant following vehicle may rapidlyapproach the subject vehicle or the subject vehicle may rapidly approacha distant preceding vehicle. Accordingly, it is necessary to performautomatic exposure control optimized for vehicle light source detectionand separation of luminous points (right and left headlights) of thedistant vehicle (or exposure control is performed with priority given tothe separation of luminous points of the distant vehicle). In contrast,on a general road or a city road, the subject vehicle and a distantvehicle are highly unlikely to approach rapidly each other. Instead,some caution should be given to vehicles and pedestrians near thesubject vehicle. In addition, most of such roads are bent and are notgood in visibility. Therefore, there is no need for the automaticexposure control optimized for vehicle light source detection andseparation of luminous points of a distant vehicle on roads other thanexpressways.

When the subject vehicle is not currently driving on an expressway (stepS1902: No), it is not necessary to shift to the vehicle light sourcedetection mode. Accordingly, the processing unit 605 returns thenegative result of determination on the shift to the vehicle lightsource detection mode (step S1905).

When the subject vehicle is currently driving on an expressway (stepS1902: Yes), the processing unit 605 further checks whether the subjectvehicle is currently driving under dark driving environments (stepS1903).

For example, the nighttime and the inside of a tunnel can be said to bedark driving environments. The processing unit 605 may determine whetherit is night time now based on time information provided by a systemclock, for example. Alternatively, the processing unit 605 may determinewhether the subject vehicle is driving in a tunnel based on the currentposition information and the map information obtained by the GPS sensor604. Alternatively, the processing unit 605 may determine whether thesubject vehicle is driving under dark driving environments such as atnight or in a tunnel, based on brightness information for the image shotby the image sensor 601.

When the subject vehicle is not driving under dark driving environments(step S1903: No), it is possible to catch accurately a distant vehicleeven by the normal exposure control under which the entire screen issubjected to average exposure control. That is, there is no need toshift to the vehicle light source detection mode, and therefore theprocessing unit 605 returns the negative result of determination on theshift to the vehicle light source detection mode (step S1905).

In contrast, when the shift to the vehicle light source detection modeis enabled and the subject vehicle is determined to be driving underdark driving environments such as at night or in a tunnel (step S1903:Yes), the processing unit 605 returns the affirmative result ofdetermination on the shift to the vehicle light source detection mode(step S1904). When determining to shift to the vehicle light sourcedetection mode through the process shown in FIG. 19, the processing unit605 performs a process for correcting the distant road vanishing pointin the driving direction of the subject vehicle in the image shot by thein-vehicle camera, at next step S704.

It has been already explained with reference to FIGS. 8 to 12 that thedistant road vanishing point in the driving direction is to be correcteddepending on curves or changes in the inclination of the road on whichthe subject vehicle is driving. The distant road vanishing point in thedriving direction herein refers to a point on which the ridge lines ofthe road converge. The vanishing point is theoretically a distant pointat infinity. The distant road vanishing point can be exactly determinedas a point where straight lines extended from optical flows (motionvectors) cross each other in the shot image.

In actuality, while the subject vehicle is about to change lanes on anexpressway, it is not necessary to check the presence of a followingvehicle at infinity. In the area behind the subject vehicle, the place apredetermined distance L (for example, L=about 150 meters) behind in thedirection of driving path of the subject vehicle may be set as anapproximate distant road vanishing point. Similarly, in the area aheadof the subject vehicle, the place the predetermined distance L ahead inthe predicted driving direction of the subject vehicle (the road surfacedirection along the road on which the subject vehicle is driving) may beset as an appropriate road vanishing point.

Referring to FIG. 20, a method for calculating an approximate distantroad vanishing point at a vehicle position 2002 the predetermineddistance L behind a current subject vehicle position 2001 on anexpressway 2000 curving gently to the right will be explained. When theroad bends an angle δθ at a spot a distance δI behind the currentvehicle position 2001, it is possible to calculate an angle θL of thebend from the vehicle position 2002 the predetermined distance L behindthe subject vehicle by integrating the angles δθ between the currentvehicle position 2001 and the vehicle position 2002 the predetermineddistance L behind the current vehicle position 2001, and approximate thedistant road vanishing point based on the distance L and the angle θL,as shown by the following mathematical expression (2). The bend angle δθin the road the distance δI behind the subject vehicle can be measuredas yaw rate of the subject vehicle by the yaw angle detection unit 602,for example. Alternatively, to approximate the distant road vanishingpoint in the area ahead of the driving direction, the bend angle δθ inthe road the distance δI ahead of the subject vehicle may be determinedbased on the current position information and the map informationobtained by the GPS sensor 504.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 2} \right\rbrack & \; \\{\theta_{L} = {\int_{0}^{L}\frac{{\delta\theta}_{l}}{\delta\; l}}} & (2)\end{matrix}$

Referring to FIG. 21, a method for calculating the approximate distantroad vanishing point at a vehicle position 2102 the predetermineddistance L behind a current subject vehicle position 2101 at a spot atthe end of an uphill 2100 will be explained. When a change in theinclination angle of the road at a spot the distance δI behind thecurrent vehicle position 2101 is designated as δϕ, it is possible tocalculate an inclination angle ϕL from the vehicle position 2102 thepredetermined distance L behind the subject vehicle by integrating thechanges δϕ in the inclination angle between the current vehicle position2101 and the vehicle position 2102 the predetermined distance L behindthe current vehicle position 2101, and approximate the distant roadvanishing point based on the distance L and the inclination angle ϕL, asshown by the following mathematical expression (3). The change δϕ in theinclination angle of the road the distance δI behind the subject vehiclecan be measured as pitch rate of the subject vehicle by the pitch angledetection unit 603, for example.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 3} \right\rbrack & \; \\{\phi_{L} = {\int_{\theta}^{L}\frac{{\delta\phi}_{l}}{\delta\; l}}} & (3)\end{matrix}$

The processing unit 605 defines a distant small frame for detectingvehicle light sources (lamps of a vehicle) based on the correcteddistant road vanishing point at next step S705. FIG. 14 illustrates anexample in which the distant small frame is defined in the simplestcase, that is, the distant small frame is defined in the area behind thesubject vehicle in the driving direction on a straight road. FIGS. 22and 23 illustrate other examples of defining the distant small frame.The image height of a camera varies depending on its optical projectionsystem. However, as far as it is not a wide-angle camera, the cameraperforms pinhole projection as central projection in many cases, and theangle of the camera is determined according to the image height of thescreen. Accordingly, the direction of road specification is decided bythe image height of the screen.

FIG. 22 illustrates an example in which the distant small frame isdefined in the area behind the subject vehicle in the driving directionduring passage through a right-hand curve road. In this case, asdescribed above in FIG. 9, a distant road vanishing point 2202 in thedriving direction is shifted leftward from a reference point 2201. Adistant small frame 2204 is preferably defined at a position shiftedslightly rightward from a position just under the distant road vanishingpoint 2202, allowing for the situations in which a distant vehicle 2203moves rightward along a road 2200 and avoiding the streetlights (notillustrated) at the sides of the road from entering in the distant smallframe. The following description is also applied to the case where thepredicted driving direction of the subject vehicle curves to the left.

FIG. 23 illustrates an example in which the distant small frame isdefined in the area behind the subject vehicle in the driving directionduring passage through a left-hand curve road. In this case, asdescribed above with reference to FIG. 10, a distant road vanishingpoint 2302 in the driving direction is shifted rightward from areference point 2301. A distant small frame 2304 is preferably definedat a position slightly leftward from a position just under the distantroad vanishing point 2302, allowing for the situation in which a distantvehicle 2303 moves leftward along a road 2300 and avoiding thestreetlights (not illustrated) at the sides of the road from entering inthe distant small frame. The following description is also applied tothe case where the predicted driving direction of the subject vehiclecurves to the left.

In the examples illustrated in FIGS. 14, 22, and 23, the rectangulardistant small frame is defined. Alternatively, the distant small framemay be defined in any other shape such as a triangle, a sector, or atrapezoid. As the ridge lines of the road become narrower toward thedistance (that is, the vanishing point) during driving, the rectangulardistant small frame contains useless regions other than the road (inother words, the regions in the absence of automobile lamps) whendefined. In contrast to this, the use of the tapered distant small frameformed in a triangular, a sector, or a trapezoidal shape can suppressthe occurrence of useless regions.

FIG. 35 illustrates an example in which a triangular distant small frame3501 is defined with reference to a distant road vanishing point 3502 inan image shot by the in-vehicle camera. First, one of the apexes of thetriangle is aligned with the distant road vanishing point 3502. Then,the two sides of the triangle are extended from the distant roadvanishing point 3502 along right and left ridge lines 3503 and 3504 ofthe road, and the triangle in that state is defined as distant smallframe 3501. The right and left ridge lines 3503 and 3504 of the road aredetected by a lane detection unit. The lane detection unit detects theridge lines of the road, lane markers and white lines near roadshoulders, and others by the use of an edge detection algorithm, anddetects the right and left ridge lines 3503 and 3504 of the road basedon the detected lines and markers. By defining the triangular distantsmall frame 3501, it is possible to prevent light sources such asstreetlights from entering into the distant small frame 3501 and detectreliably the vehicle lamps existing on the road. In addition, bydefining the triangular distant small frame 3501, it is possible toincrease the area of the distant small frame 3501 while preventing thestreetlights from entering into the distant small frame 3501.

FIG. 36 illustrates another example in which a triangular distant smallframe 3601 is defined with reference to a distant road vanishing point3602 in an image shot by the in-vehicle camera. The difference from theexample illustrated in FIG. 35 is in that the triangular distant smallframe 3601 is defined along the left lane for regions where drivers keepto the left side of the road. First, one of the apexes of the triangleis aligned with the distant road vanishing point 3602. Then, when thelane detection unit (described above) detects a left ridge line (or lanemarker near the left road shoulder) 3603 and a center line 3604, atriangle is formed such that the distant road vanishing point 3602 isset on the top and two sides of the triangle are extended along the leftridge line (or lane marker) 3603 and the center line 3604, and then thetriangle in that state is defined as distant small frame 3601. Althoughnot illustrated, for regions where drivers keep to the right side of theroad, a triangle may be formed such that the distant road vanishingpoint is set on the top and the two sides are extended along the rightridge line (or lane marker near the right road shoulder) and the centerline of the road, and the triangle in that state may be defined as adistant small frame. In this manner, by defining the triangular distantsmall frame including only the left (or right) lane, it is possible toproduce an additional merit of preventing false detection (orunnecessary detection) of the tail lamps of vehicles driving theopposite lane.

FIG. 37 illustrates an example in which a sector distant small frame3701 is defined with reference to a distant road vanishing point 3702 inan image shot by the in-vehicle camera. The example of FIG. 37 isdirected to regions where drivers keep to the left side of the road, asin the example of FIG. 36. First, the center of the sector is alignedwith the distant road vanishing point 3702. Then, the lane detectionunit (described above) detects a left ridge line (or lane marker nearthe left road shoulder) 3703 and a center line 3704 of the road, thesector is formed such that the distant road vanishing point 3702 iscentered and two radial lines are made along the left ridge line (orlane marker) 3703 and the center line 3704 of the road, and the sectorin that state is defined as distant small frame 3701. Although notillustrated, for regions where drivers keep to the right side of theroad, a sector may be formed such that the distant road vanishing pointis centered and the two radial lines are made along the right ridge line(or lane marker near the right road shoulder) and the center line of theroad, and the sector in that state may be defined as distant smallframe. In this manner, by defining not the triangular distant smallframe but the sector distant small frame, it is possible to furtherincrease the area of the distant small frame (without including uselessregions) and further enhance the probability of detection of the headlights of following vehicles (or the taillights of preceding vehicles).

FIG. 38 illustrates an example in which a trapezoidal distant smallframe 3801 is defined with reference to a distant road vanishing point3802 in an image shot by the in-vehicle camera. First, the upper andlower bases of a trapezoid are set at predetermined positions in frontof the distant road vanishing point 3802. Then, when the lane detectionunit (described above) detects right and left ridge lines of the road(or lane markers near the right and left road shoulders) 3803 and 3804,the trapezoid composed of the upper and lower bases and a pair of legsalong the right and left ridge lines 3803 and 3804 of the road isdefined as distant small frame 3801. In this manner, by defining thetrapezoidal distant small frame, it is possible to detect only the headlights of the vehicle immediately following the subject vehicle (ordetect only the taillights of the vehicle immediately preceding thesubject vehicle) to prevent false detection of the head lights ofvehicles following the immediately following vehicle (or the taillightsof vehicles preceding the immediately preceding vehicle).

FIG. 39 illustrates another example in which a trapezoidal distant smallframe 3901 is defined with reference to a distant road vanishing point3902 in an image shot by the in-vehicle camera. The difference from theexample of FIG. 38 is in that the distant small frame 3901 is definedalong the left lane for regions where drivers keep to the left side ofthe road. First, the upper and lower bases of a trapezoid are set atpredetermined positions in front of the distant road vanishing point3902. Then, when the lane detection unit (described above) detects aleft ridge line (or lane marker near left road shoulder) 3903 and acenter line 3904 of the road, the trapezoid composed of the upper andlower bases and a pair of legs along the left ridge line 3903 and thecenter line 3904 of the road is defined as distant small frame 3901.Although not illustrated, for regions where drivers keeps to the rightside of the road, the trapezoid composed of the upper and lower basesset at predetermined positions with reference to the distant roadvanishing point and a pair of legs along the right ridge line (or lanemarker near right road shoulder) and the center line of the road may bedefined as distant small frame.

FIG. 40 illustrates another example of a distant small frame definedwith reference to a distant road vanishing point 4000 in an image shotby the in-vehicle camera. In the illustrated example, defined are: afirst distant small frame 4001 composed of a sector having the centeraligned with the distant road vanishing point 4000; and an annularsecond distant small frame 4002 concentric with the first distant smallframe 4001 (in other words, having the center aligned with the distantroad vanishing point 4000). First, the center of the sector is alignedwith the distant road vanishing point 4000. Then, when the lanedetection unit (described above) detects a left ridge line (or lanemarker near the left road shoulder) 4003 and a center line 4004 of theroad, the sector is formed such that the distant road vanishing point4000 is centered and two radial lines are made along the left ridgelines (or lane marker) 4003 and the center line 4004, and the sector inthat state is defined as first distant small frame 4001. In addition,the annular second distant small frame 4002 is defined outside of thefirst distant small frame 4001, so as to have an inner perimeter and anouter perimeter at predetermined spaces from the distant road vanishingpoint 4000 and be concentric with the first distant small frame 4001.Although not illustrated, for regions where drivers keep to the rightside of the road, the first and second distant small frames may bedefined to be bilaterally symmetric with respect to the foregoing ones.

As a summary of the descriptions in FIGS. 22, 23, and 35 to 40, thedistant small frame should be defined based on the relative position ofthe distant road vanishing point in the driving direction of the subjectvehicle to the reference point (distant point at infinity in the imageshot by the in-vehicle camera). By adjusting the position of the distantsmall frame based on the relative position of the distant road vanishingpoint in the driving direction of the subject vehicle to the referencepoint, there is a merit that, even when any unnecessary light sourcessuch as the streetlights at the sides of the road on which the subjectvehicle is driving enter into the field of view of the in-vehiclecamera, the light sources can be excluded from the distant small frame.

FIG. 24 shows a flowchart of a process for defining the distant smallframe to prioritize the detection of automobile lamps executed at stepS705 in the flowchart shown in FIG. 7.

The processing unit 605 first reads the distant road vanishing pointcorrected at previous step S704 (step S2401).

Next, the processing unit 605 reads the position of the reference pointfrom the memory (ROM) 606 (step S2402). Then, the processing unit 605compares the positions of the reference point and the distant roadvanishing point (step S2403), and decides the central position of thedistant road vanishing point based on the results of comparison (stepS2404). For example, as illustrated in FIGS. 22 and 23, when the distantroad vanishing point is shifted rightward from the reference point, theprocessing unit 605 decides the central position of the distant smallframe shifted leftward from a position just under the distant roadvanishing point. In contrast, when the distant road vanishing point isshifted leftward from the reference point, the processing unit 605decides the central position of the distant small frame shiftedrightward from a position just under the distant road vanishing point.

Next, the processing unit 605 reads the size and shape of the distantsmall frame from the memory (ROM) 606 (step S2405), and decides thepositions of the starting point (upper left) and end point (lower right)of the distant small frame (step S2406). The shape of the distant smallframe is basically a landscape-oriented rectangle. However, the distantsmall frame may be changed to another shape such as a triangle, asector, or a trapezoid, depending on the position of the distant roadvanishing point or the like (see FIGS. 35 to 40).

The range of the distant small frame is preferably set such that nolight sources other than a distant vehicle such as the streetlights atthe sides of the road enter into the frame. As described above, when thedistant road vanishing point is shifted rightward from the referencepoint, the central position of the distant small frame is decided at aposition shifted leftward from a position just under the distant roadvanishing point (see FIG. 23), and in contrast, when the distant roadvanishing point is shifted leftward from the reference point, thecentral position of the distant small frame is decided at a positionshifted rightward from a position just under the distant road vanishingpoint (see FIG. 22), thereby avoiding the streetlights from enteringinto the distant small frame. In addition, the distant small frame maybe changed to another shape such as a triangle, a sector, or trapezoidso that the streetlights do not enter into the distant small frame.

Alternatively, the central position of the distant small frame may bedecided with respect to not the distant road vanishing point exactlydecided on the image shot by the in-vehicle camera but a distant roadvanishing point approximated based on a driving history of the subjectvehicle (information on changes in yaw angle and pitch angle) asdescribed above with reference to FIGS. 20 and 21.

The processing unit 605 produces a brightness horizontal profile withinthe defined distant small frame at next step S706. As described abovewith reference to FIGS. 15 to 18, the brightness horizontal profilewithin the distant small frame is produced by deciding the weights inthe individual scan lines based on the sum of brightness values of thepixels in the scan line direction, adding the weights in the individualscan lines to the brightness values of the pixels in the scan directionfor the individual horizontal pixel positions within the distant smallframe to smooth out the brightness values.

FIGS. 25A and 25B show a flowchart of a process for producing thebrightness horizontal profile within the distant small frame. In theprocess described in the drawings, brightness information for one scanline being read is recorded in the memory and the brightness informationis weighted based on the sum of brightness values in the same scan line,thereby weighting the brightness values in the scan direction.

The processing unit 605 first resets a vertical line brightness signalaccumulation memory Ysum(U) for accumulating a weighted brightness sumvalue at a horizontal pixel position U within the distant small frame, ahorizontal one-line brightness signal accumulation memory W, and ahorizontal one-line memory Ytmp(U) to their initial value of 0 (stepS2501).

Next, the processing unit 605 captures pixel data having undergoneraster scanning at preceding step S701 (step S2502).

When the scan position (U, V) falls within the range of the distantsmall frame (step S2503: Yes), the processing unit 605 adds sequentiallythe brightness Y(U,V) at that scan position to the horizontal linebrightness signal accumulation memory W (step S2504), and records thebrightness Y(U, V) in the horizontal one-line memory Ytmp(U) (stepS2508).

The processing unit 605 adds the brightness Y (U,V) with weight to thevertical line brightness signal accumulation memory Ysum(U) (stepS2505).

Next, the processing unit 605 checks whether a horizontal (U) coordinateat the scan position has reached a horizontal line last positionU=Uend(V) within the distant small frame (step S2506).

When the horizontal (U) coordinate at the scan position has not reachedthe horizontal line last position U=Uend(V) within the distant smallframe (step S2506: No), the processing unit 605 returns to step S2502 toperform repeatedly the foregoing process on the next scan data.

When the horizontal (U) coordinate at the scan position has reached thehorizontal line last position U=Uend(V) within the distant small frame(step S2506: Yes), the processing unit 605 resets both the horizontalone-line brightness signal accumulation memory W and the horizontalone-line memory Ytmp(U) to their initial value of 0 (step S2507), andreturns to step S2502 to perform repeatedly the foregoing process on thenext scan data.

In contrast, when the scan position (U, V) falls outside the range ofthe distant small frame (step S2503: No), the processing unit 605discards the signal (step S2509). Then, when the last scan position(Ulast, Vlast) within the frame has not been reached (step S2510: No),the processing unit 605 returns to step S2502 to perform repeatedly theforegoing process on the next scan data. When the last scan position(Ulast, Vlast) within the frame has been reached (step S2510: Yes), thebrightness horizontal profile (see FIG. 18) is completed and the routineof this process is terminated.

FIGS. 26A and 26B show a flowchart of another process for producing thebrightness horizontal profile within the distant small frame. In theprocess shown in the drawings, unlike in the process shown in FIGS. 25Aand 25B, weighting is performed based on the brightness sum value in onepreceding scan line, and the brightness values are weighted in the scandirection. This process saves memory capacity for recording brightnessinformation for one scan line.

The processing unit 605 first resets the vertical line brightness signalaccumulation memory Ysum(U) for accumulating a weighted brightness sumvalue at the horizontal pixel position U within the distant small frame,the horizontal one-line brightness signal accumulation memory W, and thehorizontal one-line memory W0 to their initial value of 0 (step S2601).

Next, the processing unit 605 captures pixel data having undergoneraster scanning at preceding step S701 (step S2602).

When the scan position (U, V) falls within the range of the distantsmall frame (step S2603: Yes), the processing unit 605 adds sequentiallythe brightness Y(U,V) at that scan position to the horizontal linebrightness signal accumulation memory W (step S2604).

The processing unit 605 adds the brightness Y(U,V) with the weightrecorded in the horizontal one-line brightness signal accumulationmemory W0 to the vertical line brightness signal accumulation memoryYsum(U) (step S2605). The memory W0 records the weight based on thebrightness sum value in one preceding scan line.

Next, the processing unit 605 checks whether a horizontal (U) coordinateat the scan position has reached a horizontal line last positionU=Uend(V) within the distant small frame (step S2606).

When the horizontal (U) coordinate at the scan position has not reachedthe horizontal line last position U=Uend(V) within the distant smallframe (step S2606: No), the processing unit 605 returns to step S2602 toperform repeatedly the foregoing process on the next scan data.

When the horizontal (U) coordinate at the scan position has reached thehorizontal line last position U=Uend(V) within the distant small frame(step S2506: Yes), the processing unit 605 substitutes the value in thehorizontal one-line brightness signal accumulation memory W in thememory W0 and resets the memory to its initial value of 0 (step S2607),and then returns to step S2602 to perform repeatedly the process on thenext scan data.

In contrast, when the scan position (U, V) falls outside the range ofthe distant small frame (step S2603: No), the processing unit 605discards the signal (step S2608). Then, when the last scan position(Ulast, Vlast) within the frame has not been reached (step S2609: No),the processing unit 605 returns to step S2602 to perform repeatedly theforegoing process on the next scan data. When the last scan position(Ulast, Vlast) within the frame has been reached (step S2609: Yes), thebrightness horizontal profile (see FIG. 18) is completed and the routineof this process is terminated.

The processing unit 605 checks at next step S707 whether there are thelamps of an automobile within the distant small frame (that is, theheadlights of a distant oncoming vehicle (or following vehicle) or thetaillights of a following vehicle)) within the distant small frame basedon the brightness horizontal profile. As described above with referenceto FIG. 18, it is expected that two peaks (maximum points) appear in thebrightness horizontal profile within the distant small frame includingthe two luminous points (the headlights of the distant vehicle). Asillustrated in FIG. 27, it is possible to determine easily on thepresence or absence of the lamps of an automobile within the distantsmall frame, based on brightness information and positional relationshiprelating to adjacent maximum values 2701 and 2702 and a minimum value2703 between these maximum values in a brightness horizontal profile2700 within the distant small frame.

FIGS. 28A and 28B show a flowchart of a detailed process for determiningwhether there are the lamps of an automobile within the distant smallframe based on the brightness horizontal profile.

The processing unit 605 first reads the brightness horizontal profileproduced at previous step S706 (step S2801). Then, the processing unit605 analyzes the read brightness horizontal profile to detect the twoadjacent maximum values, the minimum value between the maximum values,and their positions (horizontal pixel positions) (step S2802).

Next, the processing unit 605 calculates the interval between Xcoordinates of the two adjacent maximum values and the average of thesame (step S2803). Then, the processing unit 605 checks whether theinterval between the X coordinates of the adjacent maximum values fallswithin a predetermined range (step S2804).

When the interval between the X coordinates of the adjacent maximumvalues falls outside the predetermined range (step S2804: No), theprocessing unit 605 determines that luminous points corresponding to thetwo maximum values are not the lamps (headlights or taillights) of adistant vehicle and returns the result of determination that there areno lamps of an automobile within the distant small frame (step S2808).Besides the lamps of an automobile, the maximum values may indicate thestreetlights at the sides of the road or illumination of buildingsaround the road as an example.

In contrast, when the interval between the X coordinates of the adjacentmaximum values falls within the predetermined range (step S2804: Yes),the processing unit 605 further checks whether the average of theadjacent maximum values is a predetermined times larger than the minimumvalue between the maximum values or more (step S2805).

When the average of the adjacent maximum values is the predeterminedtimes larger than the minimum value between the maximum values or more(step S2805: Yes), the two adjacent maximum values can be said to be twoluminous points (headlights or taillights). The processing unit 605returns the result of determination that there are the lamps of anautomobile within the distant small frame (step S2806).

When the average of the adjacent maximum values is the predeterminedtime larger than the minimum value or more (step S2805: No), theprocessing unit 605 further checks whether the average of the adjacentmaximum values is equal to or more than a predetermined value (stepS2807).

When the average of the adjacent maximum values is not the predeterminedtime larger than the minimum value between the adjacent maximum valuesor more but the average of the adjacent maximum values is equal to ormore than the predetermined value (step S2807: Yes), the adjacentmaximum values can be said to be two luminous points (headlights ortaillights). In this case, the processing unit 605 returns the result ofdetermination that there are the lamps of an automobile within thedistant small frame (step S2806).

When the average of the adjacent maximum values is not equal to or morethan the predetermined value (step S2807: No), the processing unit 605determines that the luminous points corresponding to the two maximumvalues are not the lamps (headlights or taillights) of a distant vehicleand returns the result of determination that there are no lamps of anautomobile within the distant small frame (step S2808).

When returning the result of determination that there are no lamps of anautomobile within the distant small frame, the processing unit 605controls an exposure of the in-vehicle camera in the normal automaticexposure control mode. When returning the result of determination thatthere are the lamps of an automobile within the distant small frame, theprocessing unit 605 applies the automatic exposure control optimizedsuch that the two luminous points are separated from the brightnesshorizontal profile (or such that exposure control is performed withpriority given to separation of luminous points of a distant vehicle) tothe in-vehicle camera.

When determining that there are the lamps of an automobile within thedistant small frame, the processing unit 605 applies the automaticexposure control optimized such that the two luminous points can beseparated in the brightness horizontal profile (or separation of theluminous points is prioritized) as a process at step S710 to thein-vehicle camera.

Describing with the example illustrated in FIG. 27, the luminous pointseparation optimized automatic exposure control is a process forallowing easy separation of the adjacent maximum values 2701 and 2702with the minimum value 2703 between the maximum values in the brightnesshorizontal profile 2700 within the distant small frame.

FIG. 29 shows a flowchart of a process for performing automatic exposurecontrol of the in-vehicle camera optimized for luminous point separation(or with priority given to luminous point separation). The aim of theautomatic exposure control is to allow the shape of the headlights tosatisfy the foregoing mathematic expression (1) when the in-vehiclecamera shots a following vehicle or an oncoming vehicle in the sceneunder dark driving environments.

The processing unit 605 first reads the brightness horizontal profileproduced at preceding step S706 (step S2901).

Next, the processing unit 605 analyzes the read brightness horizontalprofile to detect the two adjacent maximum values and the minimum valuebetween the maximum values (step S2902). Alternatively, the processingunit 605 may read directly the maximum values and the minimum valuedetected in the process within the distant small frame performed at stepS707.

Next, the processing unit 605 divides the average of the adjacentmaximum values by the minimum value to calculate an evaluation value Ffor the automatic exposure control (step S2903).

Next, the processing unit 605 compares the calculated evaluation value Fto a predetermined threshold value Fth (step S2904). The threshold valueFth is stored within the memory (ROM) 606, the processing unit 605 readsthe threshold value Fth from the memory 606 at step S2904.

When the evaluation value F is larger than the threshold value Fth (stepS2905: Yes), the minimum value is small as illustrated in FIG. 30A. Thisis possibly because the light exposure of the in-vehicle camera is toolow, and it is presumed that the shot image appears entirely dark, andthe luminous points corresponding to the lamps (headlights ortaillights) of a distant vehicle are darkly seen. Thus, the processingunit 605 increases the exposure time or the gain of the in-vehiclecamera (step S2906).

In contrast, when the evaluation value F is not larger than thethreshold value Fth (step S2905: No), the minimum value is large. In thebrightness horizontal profile, the difference between the maximum valuesand the minimum value is small as illustrated in FIG. 30B. This ispossibly because the light exposure of the in-vehicle camera is toohigh, and the luminous points corresponding to the lamps (headlights ortaillights) of the distant vehicle appear blurred and their existence isobscure. Thus, the processing unit 605 decreases the exposure time orthe gain of the in-vehicle camera (step S2907).

The threshold value Fth may be a fixed value or a threshold range with apredetermined width. For example, a threshold value Fth1 and a thresholdvalue Fth2 smaller than the threshold value Fth1 may be set, and theexposure time or the gain of the in-vehicle camera may be decreased whenthe evaluation value F is larger than the threshold value Fth1, and theexposure time or the gain of the in-vehicle camera may be increased whenthe evaluation value F is smaller than the threshold value Fth2.However, the threshold value Fth1 is desirably 1.1 or more.

As described above with reference to FIG. 18, two peaks (maximum points)appear in the brightness horizontal profile in the distant small frameincluding the two luminous points (the headlights of a distant vehicle).Accordingly, when shooting is performed in the exposure condition (theexposure time or the gain) optimized for (or prioritizing) separation ofthe two luminous points of the distant vehicle, it is expected that, ina brightness horizontal profile 3100 within the distant small frame, awidth maximum value Cmax of a peak exceeding a predetermined thresholdvalue (brightness comparison parameter) Fref is larger than apredetermined target value as illustrated in FIG. 31. When the lightexposure is insufficient (the exposure time or the gain is small), it ispresumed that, in the brightness horizontal profile 3100 within thedistant small frame, the width maximum value Cmax of the peak exceedingthe predetermined threshold value (brightness comparison parameter) Frefis smaller than the predetermined target value as illustrated in FIG.32. Therefore, the luminous point separation optimized automaticexposure control may be performed based on the shape of the brightnesshorizontal profile within the distant small frame.

FIG. 33 shows a flowchart of a process for performing automatic exposurecontrol of the in-vehicle camera optimized for (or prioritizing)luminous point separation, based on the shape of the brightnesshorizontal profile within the distant small frame. The aim of theautomatic exposure control is to allow the shape of the headlights tosatisfy the foregoing mathematical expression (1) when a followingvehicle or an oncoming vehicle is shot by the in-vehicle camera in thescene under dark driving environments.

The processing unit 605 first reads the brightness horizontal profileproduced at preceding step S706 (step S3301).

Next, the processing unit 605 resets the memories to their initial value(step S3302). Specifically, the processing unit 605 substitutes astarting point horizontal position Ustart within the distant small frameinto a horizontal position memory X. The processing unit 605 reads theviewpoint position in the distant small frame from the result ofdefinition of the distant small frame at preceding step S705. Theprocessing unit 605 also resets both a width counter Cport and a peakwidth maximum value memory Cmax to their initial value of 0.

Next, the processing unit 605 compares a weighted brightness sum valueF(x) at a current horizontal position X within the distant small frameto the predetermined threshold value (brightness comparison parameter)Fref (step S3303). The brightness comparison parameter Fref is stored inthe memory (ROM) 606. The processing unit 605 reads the brightnesscomparison parameter Fref from the memory 606 at step S3303.

When the weighted brightness sum value F(x) is larger than thebrightness comparison parameter Fref (step S3303: Yes), the currenthorizontal position X corresponds to a peak portion in the brightnesshorizontal profile. The processing unit 605 updates the width counterCport (step S3304) (Cport=Cport+1). Then, the processing unit 605compares the width counter to the width maximum value Cmax (step S3305).When the width Cport of the peak being counted is larger than the widthmaximum value Cmax (step S3305: Yes), the processing unit 605 updatesthe width maximum value Cmax by the width Cport (step S3306).

Upon completion of the process with the current horizontal position X inthis manner, the processing unit 605 updates X (step S3307) (X=X+1), andchecks whether an end point horizontal position Uend within the distantsmall frame has been reached, that is, whether the process on the entirebrightness horizontal profile is completed (step S3308).

When the end point horizontal position Uend within the distant smallframe has not yet been reached (step S3308: No), the processing unit 605returns to step S3303 to perform repeatedly the foregoing process on thenext horizontal position X within the distant small frame.

In contrast, when the end point horizontal position Uend in the distantsmall frame has been reached (step S3308: Yes), the processing unit 605compares the width maximum value Cmax of the peak in the brightnesshorizontal profile to a predetermined target value Ctarget (step S3309).The target value Ctarget is stored in the memory (ROM) 606, and theprocessing unit 605 reads the target value Ctarget from the memory 606at step S3303.

When the peak width maximum value Cmax is larger than the target valueCtarget (step S3309: Yes), the two luminous points corresponding to thelamps (headlights or taillights) of the distant vehicle clearly appearas maximum values in the brightness horizontal profile as illustrated inFIG. 31. Accordingly, to avoid overexposure, the processing unit 605decreases the exposure time or the gain of the in-vehicle camera (stepS3310). Alternatively, the processing unit 605 may make some adjustmentsthrough the use of iris diaphragm, variable ND filter, or tonecorrection.

In contrast, when the peak width maximum value Cmax is not larger thanthe target value Ctarget (step S3309: No), the difference between themaximum values and the minimum value is small in the brightnesshorizontal profile as illustrated in FIG. 32 and the existence of theluminous points is obscure. This is possibly because the light exposureof the in-vehicle camera is too low, and the luminous pointscorresponding to the lamps (headlights or taillights) of the distantvehicle appear dark. Thus, the processing unit 605 increases theexposure time or the gain of the in-vehicle camera (step S3311).

The in-vehicle camera related to driving support and distant visualfield support needs to catch more correctly distant vehicles possiblyapproaching the subject vehicle. However, under dark drivingenvironments such as at night or in a tunnel, when the lamps ofhigh-illumination (backlight) surrounding vehicles are shot by thein-vehicle camera, the proportion of the headlights of the distantvehicle as light sources in the viewing field is small in particular.Accordingly, when the normal automatic light exposure control is appliedto perform an average exposure control on the entire screen,overexposure is likely to occur to saturate even peripheral pixels inwhite, and it is difficult to catch the headlights as two luminouspoints.

It can be said that the distant vehicle possibly approaching the subjectvehicle is driving in the direction of the driving path of the subjectvehicle in the area behind the subject vehicle, or is driving in thepredicted driving direction of the subject vehicle (the road surfacedirection along the road on which the subject vehicle is driving) in thearea ahead of the subject vehicle.

As described above, the technology disclosed herein is intended toperform automatic exposure control of the in-vehicle camera only withthe brightness information within the distant small frame defined withreference to the distant road vanishing point in the driving directionof the subject vehicle. It is expected that the in-vehicle camerashooting the area behind the subject vehicle catches a distant vehicledriving in the direction of the driving path within the distant smallframe, and the in-vehicle camera shooting the area ahead of the subjectvehicle catches a distant vehicle driving in the predicted drivingdirection of the subject vehicle within the distant small frame.Therefore, by analyzing the brightness information only within thedistant small frame to perform exposure control of the in-vehiclecamera, it is possible to shoot the distant vehicle with the headlightsilluminated under dark driving environments while avoiding overexposure.

The technology disclosed herein is intended to sum up the brightnessvalues of pixels in one each horizontal line within the distant smallframe to decide weights in the individual horizontal lines, and add theweights in the horizontal lines to the brightness values of the pixelsin the individual vertical lines to produce a brightness horizontalprofile, and perform exposure control of the in-vehicle camera based onthe result of analysis of the brightness horizontal profile. Therefore,by producing the brightness horizontal profile only within the distantsmall frame, it is possible to reproduce the target light sources on apriority basis in an efficient manner with the small memory. As aresult, it is possible to shoot the lamps of an automobile (theilluminated headlights of the distant vehicle at night or the like)within the distant small frame in a preferable manner while avoidingoverexposure.

INDUSTRIAL APPLICABILITY

The technology disclosed herein has been explained in detail withreference to the specific embodiments. However, it is obvious thatpersons skilled in the art can modify the embodiments or replace thesame by any others without deviating from the gist of the technologydisclosed herein.

The technology disclosed herein can be applied to automatic exposurecontrol of the in-vehicle camera under dark driving environments such asat night or in a tunnel to shoot high-illuminous lamps (luminous points)such as the headlights or taillights of a distant vehicle while avoidingoverexposure.

The technology disclosed herein is not limited to standard-sizedautomobiles but may be applied to various types of vehicles equippedwith in-vehicle cameras such as medium-sized automobiles, large-sizedautomobiles, and motorcycles.

Briefly, although the technology disclosed herein has been explainedtaking some examples, the descriptions therein should not be interpretedin a limited way. The claims should be referred to determine the gist ofthe technology disclosed herein. The technology disclosed herein canhave the following configurations:

(1)

A vehicle camera system including:

a vehicle camera mounted in a vehicle configured to capture surroundingsof the vehicle; and

control circuitry configured to control an exposure level of an imagecaptured by the vehicle camera, the control of the exposure level beingbased on brightness information of a detection area set within theimage, the detection area being a portion of the captured image andconfigured to output the image having exposure control performed thereonto a display.

(2)

The vehicle camera system according to (1), wherein the controlcircuitry is further configured to control the exposure level of theimage captured by the vehicle camera such that light sources of anothervehicle in the detection area are displayed separately within the imageby the display.

(3)

The vehicle camera system according to (1)-(2), wherein the controlcircuitry is further configured to determine whether to perform theexposure control depending on settings of the vehicle camera system.

(4)

The vehicle camera system according to (1)-(3), wherein the controlcircuitry is configured to determine whether to perform the exposurecontrol depending on a type of road on which the vehicle is driving.

(5)

The vehicle camera system according to (1)-(4), wherein the controlcircuitry is further configured to determine whether to perform theexposure control depending on environmental conditions present while thevehicle is driving.

(6)

The vehicle camera system according to (1)-(5), wherein

the control circuitry is further configured to set the detection areawith reference to a vanishing point in the image, and perform theexposure control based on the brightness information within thedetection area.

(7)

The vehicle camera system according to (1)-(6), wherein the controlcircuitry is further configured to correct the vanishing point based ona course of the vehicle driving path in the area behind the vehicle.

(8)

The vehicle camera system according to (1)-(7), wherein the controlcircuitry is configured to set the detection area based on a relativeposition of the vanishing point to a reference point in the imagecaptured by the vehicle camera.

(9)

The vehicle camera system according to (1)-(8), wherein the controlcircuitry is configured to perform exposure control on the imagecaptured by the vehicle camera based on the brightness information whichincludes a result of a brightness horizontal profile produced within thedetection area.

(10)

The vehicle camera system according to (1)-(9), wherein the controlcircuitry is configured to sum up brightness values of pixels in eachhorizontal line within the detection area in order to determine weightsfor the respective individual horizontal lines, and to add the weightsfor the respective individual horizontal lines to the brightness valuesof pixels in order to produce the brightness horizontal profile.

(11)

The vehicle camera system according to (1)-(10), wherein the controlcircuitry is configured to detect whether there are lights of anothervehicle within the detection area, and when detecting the presence ofthe lights of the another vehicle within the detection area, control theexposure level of the image captured by the vehicle camera such that thelights of the another vehicle are displayed separately on the display.

(12)

The vehicle camera system according to (1)-(11), wherein the controlcircuitry is configured to detect whether the lights of the anothervehicle are within the detection area based on brightness information ofa minimum value between two adjacent maximum values included in thebrightness horizontal profile.

(13)

The vehicle camera system according to (1)-(12), wherein, when aninterval between the adjacent maximum values falls within apredetermined range and an average of the adjacent maximum values is apredetermined number of times larger than the minimum value, the controlcircuitry is configured to determine that the lights of the anothervehicle are within the detection area.

(14)

The vehicle camera system according to (1)-(12), wherein when theinterval between the adjacent maximum values falls within apredetermined range and the average of the adjacent maximum values ismore than a predetermined value, the control circuitry is configured todetermine that the lights of the another vehicle are within thedetection area.

(15)

The vehicle camera system according to (1)-(14), further comprising analarm, wherein when the exposure control is being performed, the controlcircuitry is configured to notify a driver in advance with the alarm.

(16)

The vehicle camera system according to (1)-(15), wherein the controlcircuitry is configured to perform exposure control on the imagecaptured by the vehicle camera such that lights of another vehicle aredisplayed separately on the display, the control circuitry performingthe exposure control based on a brightness horizontal profile producedwithin the detection area.

(17)

The vehicle camera system according to (1)-(16), wherein the controlcircuitry is configured to perform exposure control on the imagecaptured by the vehicle camera based on an evaluation value determinedby dividing an average of two maximum values by a minimum value disposedbetween the maximum values in the brightness horizontal profile.

(18)

The vehicle camera system according to (1)-(16), wherein the controlcircuitry is configured to perform exposure control on the imagecaptured by the vehicle camera based on a distance of peaks of maximumvalues in the brightness horizontal profile.

(19)

The vehicle camera system according to (1)-(18), wherein the controlcircuitry is further configured to control exposure level of thedetection area of the image captured by the vehicle camera.

(20)

An image processing apparatus, including:

control circuitry configured to control an exposure level of an imagecaptured by a vehicle camera, the control of the exposure level beingbased on brightness information of a detection area set within theimage, the detection area being a portion of the captured image andconfigured to output the image having exposure control performed thereonto a display.

(21)

An image processing method, including:

controlling, using control circuitry, an exposure level of an imagecaptured by a vehicle camera, the control of the exposure level beingbased on brightness information of a detection area set within theimage, the detection area being a portion of the captured image; and

outputting the image having exposure control performed thereon to adisplay.

(22)

An in-vehicle camera system comprising:

an in-vehicle camera that is mounted in a vehicle to shoot surroundingsof the vehicle; and

a control unit that controls exposure of the vehicle based on brightnessinformation of a detection area set within an image shot by thein-vehicle camera.

(23)

The in-vehicle camera system according to (22), wherein the control unitshifts to an exposure control mode that prioritizes separation of lightsources of a vehicle other than the vehicle depending on the result ofdetection of the light sources within the detection area.

(24)

The in-vehicle camera system according to (23), wherein the control unitdetermines whether to perform the detection of the light sourcesdepending on settings in the in-vehicle camera system.

(25)

The in-vehicle camera system according to (24), wherein the control unitdetermines whether to perform the detection of the light sourcesdepending on the type of the road on which the vehicle is driving.

(26)

The in-vehicle camera system according to (23), wherein the control unitdetermines whether to perform the detection of the light sourcesdepending on driving environments of the vehicle.

(27)

The in-vehicle camera system according to (22), wherein

the control unit sets the detection area with reference to a distantroad vanishing point in a driving direction of the vehicle, and performsexposure control of the in-vehicle camera based on the result ofanalysis of brightness information within the detection area.

(28)

The in-vehicle camera system according to (27), wherein the control unitcorrects the distant road vanishing point based on a direction of adriving path of the vehicle in the area behind the vehicle, or correctsthe distant road vanishing point based on a predicted driving directionof the vehicle in the area ahead of the vehicle.

(29)

The in-vehicle camera system according to (27), wherein the control unitdetermines a place a predetermined distance behind the vehicle in thedirection of the driving path of the vehicle as an approximate distantroad vanishing point, or determines a place the predetermined distanceahead of the vehicle in a road surface direction along the road on whichthe vehicle is driving as an approximate distant road vanishing point.

(30)

The in-vehicle camera system according to (27), wherein the control unitsets the detection area based on a relative position of the distant roadvanishing point in the direction of the vehicle to a distant point atinfinity in the image shot by the in-vehicle camera.

(31)

The in-vehicle camera system according to (27), wherein the control unitperforms exposure control of the in-vehicle camera based on the resultof analysis of a brightness horizontal profile produced within thedetection area as brightness information within the detection area.

(32)

The in-vehicle camera system according to (31), wherein the control unitsums up brightness values of pixels at least in one each horizontal linewithin the detection area to decide weights in the individual horizontallines, and adds the weights in the horizontal lines to the brightnessvalues of pixels in individual vertical lines to produce the brightnesshorizontal profile.

(33)

The in-vehicle camera system according to (27), wherein the control unitdetects whether there are the lamps of the other vehicle within thedetection area, and when detecting the presence of the lamps of theother vehicle within the detection area, controls exposure of thein-vehicle camera with priority given to separation of luminous pointsof the lamps of the other vehicle.

(34)

The in-vehicle camera system according to (33), wherein the control unitdetects whether there are the lamps of the other vehicle within thedetection area, based on brightness information of a minimum valuebetween two adjacent maximum values included in the brightnesshorizontal profile.

(35)

The in-vehicle camera system according to (34), wherein, when theinterval between the maximum values falls within a predetermined rangeand the average of the maximum values is predetermined times larger thanthe minimum value or more or the average of the maximum values is equalto or more than a predetermined value, the control unit determines thatthere are the lamps of the other vehicle within the detection area.

(36)

The in-vehicle camera system according to (27), wherein the control unitdecides an exposure control mode for the in-vehicle camera based onwhether the lamps of the other vehicle are detected within the detectionarea.

(37)

The in-vehicle camera system according to (36), further comprising animage output unit or a sound output unit, wherein when changing theexposure control mode, the control unit notifies of that in advance withthe use of at least one of an image, a sound, and a vibration.

(38)

The in-vehicle camera system according to (22), wherein the control unitperforms exposure control of the in-vehicle camera with priority givento separation of luminous points of the lamps of the other vehicle,based on a brightness horizontal profile produced within the detectionarea.

(39)

The in-vehicle camera system according to (38), wherein the control unitperforms exposure control of the in-vehicle camera, based on anevaluation value determined by dividing the average of two adjacentmaximum values by a minimum value between the maximum values included inthe brightness horizontal profile.

(40)

The in-vehicle camera system according to (38), wherein the control unitperforms exposure control of the in-vehicle camera, based on a maximumvalue of a peak width exceeding a predetermined threshold value in thebrightness horizontal profile.

(41)

An image processing apparatus that controls exposure of an in-vehiclecamera mounted in a vehicle, based on brightness information in adetection area set within an image shot by the in-vehicle camera.

REFERENCE SIGNS LIST

-   600 In-vehicle camera system-   601 Image sensor-   602 Yaw angle detection unit-   603 Pitch angle detection unit-   604 GPS sensor-   605 Processing unit-   606 Memory-   607 Setting input unit-   608 Image output unit-   609 Sound output unit

The invention claimed is:
 1. A vehicle camera system, comprising:control circuitry; and an imaging sensor mounted in a first vehicle,wherein the imaging sensor is configured to: capture an image ofsurroundings of the first vehicle; and output the captured image to thecontrol circuitry, the control circuitry is configured to: determine,for each horizontal line within a detection area of the captured image,a respective specific brightness value based on addition of brightnessvalues of pixels in each horizontal line within the detection area ofthe captured image; determine a corresponding weight for each horizontalline based on the respective specific brightness value; generate abrightness horizontal profile based on addition of the correspondingweight for each horizontal line to the brightness values of the pixelsin each horizontal line; and execute an exposure control process basedon the generated brightness horizontal profile, the generated brightnesshorizontal profile is based on a portion of the captured image, and theportion of the captured image corresponds to the detection area withinthe captured image to separately display lights of a second vehicle on adisplay screen.
 2. The vehicle camera system according to claim 1,wherein the control circuitry is further configured to: set thedetection area with reference to a vanishing point in the capturedimage; and execute the exposure control process based on the brightnesshorizontal profile within the detection area.
 3. The vehicle camerasystem according to claim 2, wherein the control circuitry is furtherconfigured to correct the vanishing point based on a course of a vehicledriving path in an area behind the first vehicle.
 4. The vehicle camerasystem according to claim 2, wherein the control circuitry is furtherconfigured to set the detection area based on a relation of a positionof the vanishing point with a reference point in the captured image. 5.The vehicle camera system according to claim 1, wherein the exposurecontrol process comprises at least one of exposure time control, gaincontrol, or a combination thereof.
 6. The vehicle camera systemaccording to claim 1, wherein the control circuitry is furtherconfigured to: detect the lights of the second vehicle within thedetection area; and execute the exposure control process based on apresence of the lights of the second vehicle within the detection area,wherein the lights of the second vehicle are displayed separately on thedisplay screen.
 7. The vehicle camera system according to claim 6,wherein the control circuitry is further configured to detect the lightsof the second vehicle within the detection area based on a local minimumvalue between two adjacent maximum values in the brightness horizontalprofile.
 8. The vehicle camera system according to claim 7, wherein thecontrol circuitry is further configured to determine the lights of thesecond vehicle within the detection area based on an interval betweenthe two adjacent maximum values and an average of the two adjacentmaximum values, the interval is within a specific range, and the averageis a specific number of times larger than the local minimum value. 9.The vehicle camera system according to claim 7, wherein the controlcircuitry is further configured to determine the lights of the secondvehicle within the detection area based on an interval between the twoadjacent maximum values and an average of the adjacent maximum values,the interval is within a specific range, and the average is more than aspecific value.
 10. The vehicle camera system according to claim 1,wherein the control circuitry is further configured to: determine anevaluation value based on a division of an average of two adjacentmaximum values by a local minimum value in the brightness horizontalprofile, wherein the local minimum value is between the two adjacentmaximum values; and execute the exposure control process on the capturedimage based on the evaluation value.
 11. The vehicle camera systemaccording to claim 1, wherein the control circuitry is furtherconfigured to execute the exposure control process on the captured imagebased on a distance of peaks of two adjacent maximum values in thebrightness horizontal profile.
 12. The vehicle camera system accordingto claim 1, wherein the control circuitry is further configured toexecute the exposure control process based on settings of the vehiclecamera system.
 13. The vehicle camera system according to claim 1,wherein the control circuitry is further configured to execute theexposure control process based on environmental conditions associatedwith the surroundings of the first vehicle.
 14. The vehicle camerasystem according to claim 1, wherein the control circuitry is configuredto execute the exposure control process based on a type of road on whichthe first vehicle is driven.
 15. The vehicle camera system according toclaim 1, further comprising an alarm, wherein in the exposure controlprocess, the control circuitry is further configured to notify a driverwith the alarm.
 16. A vehicle camera system, comprising: controlcircuitry; and an imaging sensor mounted in a first vehicle, wherein theimaging sensor is configured to: capture an image of surroundings of thefirst vehicle; and output the captured image to the control circuitry,the control circuitry is configured to: determine an evaluation valuebased on division of a maximum value in a brightness horizontal profileby a local minimum value in the brightness horizontal profile; andexecute an exposure control process based on the determined evaluationvalue, the local minimum value is between two adjacent local maximumvalues in the brightness horizontal profile, the brightness horizontalprofile is based on a portion of the captured image corresponding to adetection area within the captured image, and lights of a second vehicleare displayed separately on a display screen.
 17. The vehicle camerasystem according to claim 16, wherein the exposure control processcomprises at least one of exposure time control, gain control, or acombination thereof.
 18. The vehicle camera system according to claim17, wherein the control circuitry is further configured to increase atleast one of an exposure time associated with the imaging sensor or again associated with the imaging sensor, and the at least one of theexposure time or the gain is increased based on the evaluation valuethat is larger than a specific threshold.
 19. The vehicle camera systemaccording to claim 17, wherein the control circuitry is furtherconfigured to decrease at least one of an exposure time associated withthe imaging sensor or a gain associated with the imaging sensor, and theat least one of the exposure time or the gain is decreased based on theevaluation value that is smaller than a specific threshold.